Branching on Attribute Values in Decision Tree Generation

The problem of deciding which subset of values of a categorical-valued attribute to branch on during decision tree generation is addressed. Algorithms such as ID3 and C4 do not address the issue and simply branch on each value of the selected attribute. The GID3* algorithm is presented and evaluated. The GID3* algorithm is a generalized version of Quinlan’s ID3 and C4, and is a non-parametric version of the GID3 algorithm presented in an earlier paper. It branches on a subset of individual values of an attribute, while grouping the rest under a single DEFAULT branch. It is empirically demonstrated that GID3* outperforms ID3 (C4) and GID3 for any parameter setting of the latter. The empirical tests include both controlled synthetic (randomized) domains as well as real-world data sets. The improvement in tree quality as measured by number of leaves and estimated error rate is significant. Introduction Empirical learning algorithms attempt to discover relations between situations expressed in terms of a set of attributes and actions encoded in terms of a fixed set of classes. By examining large sets of pre-classified data, it is hoped that a learning program may discover the proper conditions under which each action (class) is appropriate. Heuristic methods are used to perform guided search through the large space of possible relations between combinations of attribute values and classes. A powerful and popular such heuristic uses the notion of selecting attributes that locally minimize the information entropy of the classes in a data set. This heuristic is used in the ID3 algorithm [l l] and its extensions, e.g. GID3 [2], GID3* [4], and C4 [12], in CART [l], in CN2 [3] and others; see [4, 5, lo] for a general discussion of the attribute selection problem. The attributes in a learning problem may be discrete (categorical), or they may be continuous (numerical). The above mentioned attribute selection process assumes that all attributes are discrete. Continuous-valued attributes must, therefore, be discretized prior to attribute selection. This is typically achieved by partitioning the range of the attribute into subranges, i.e., a test is devised that quantizes the range. In this paper, we focus only on the problem of deciding which values of a discrete-valued (or discretized) attribute should be branched on, and which should not. We propose that by avoiding branching on all values (as in ID3), better trees are obtained. We originally developed the GID3 algorithm [2] to address this problem. GID3 is dependent on a user-determined parameter setting (TL) that controls its tendency towards branching on some versus all values of an attribute. We have demonstrated that for certain settings of TL, GID3 produces significantly better trees than ID3 or C4l. In this paper we present the GID3* algorithm in which the dependence on a user-specified parameter has been removed. We empirically demonstrate that GID3* produces better trees than GID3 for a wide range of parameter settings. The Attribute Selection Criterion Assume we are to select an attribute for branching at a node having a set S of N examples from a set of k classes: {Cl,..., Ck}. Assuming that some test T on attribute A partitions the set S into the subsets St, . . . , S,. Let P(Ci, S) be the proportion of examples in S that have class C,:. The class e&&y of a subset S is defined as: Ent(S) = 2 P(ci, S) log(p(ci, s>>